Computer system and process for transferring multiple high bandwidth streams of data between multiple storage units and multiple applications in a scalable and reliable manner

ABSTRACT

Multiple applications request data from multiple storage units over a computer network. The data is divided into segments and each segment is distributed randomly on one of several storage units, independent of the storage units on which other segments of the media data are stored. At least one additional copy of each segment also is distributed randomly over the storage units, such that each segment is stored on at least two storage units. This random distribution of multiple copies of segments of data improves both scalability and reliability. When an application requests a selected segment of data, the request is processed by the storage unit with the shortest queue of requests. Random fluctuations in the load applied by multiple applications on multiple storage units are balanced nearly equally over all of the storage units. This combination of techniques results in a system which can transfer multiple, independent high-bandwidth streams of data in a scalable manner in both directions between multiple applications and multiple storage units.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. §120, and is acontinuation of: U.S. patent application Ser. No. 13/932,225, filed onJul. 1, 2013, now pending, which is a continuation of U.S. patentapplication Ser. No. 13/423,778, filed on Mar. 19, 2012, now issued asU.S. Pat. No. 8,478,957, which is a continuation of U.S. patentapplication Ser. No. 13/070,581, filed on Mar. 24, 2011, now issued asU.S. Pat. No. 8,140,755, which is a continuation of U.S. patentapplication Ser. No. 12/689,426, filed on Jan. 19, 2010, now issued asU.S. Pat. No. 7,917,696, which is a continuation of U.S. patentapplication Ser. No. 11/986,226, filed on Nov. 20, 2007, now issued asU.S. Pat. No. 7,660,947, which is a continuation of U.S. patentapplication Ser. No. 11/522,758, filed on Sep. 18, 2006, now issued asU.S. Pat. No. 7,487,309, which is a continuation of U.S. patentapplication Ser. No. 10/883,387, filed on Jul. 1, 2004, now issued asU.S. Pat. No. 7,111,115, which is a continuation of U.S. patentapplication Ser. No. 10/186,929, filed on Jul. 1, 2002, now issued asU.S. Pat. No. 6,760,808, which is a continuation of U.S. patentapplication Ser. No. 09/006,070, filed on Jan. 12, 1998, now issued asU.S. Pat. No. 6,415,373, which is a continuation of U.S. patentapplication Ser. No. 08/997,769, filed on Dec. 24, 1997, now abandoned,all of which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention is related to computer systems for capture,authoring and playback of multimedia programs and to distributedcomputing systems.

BACKGROUND OF THE INVENTION

There are several computer system architectures that support distributeduse of data over computer networks. These computer system architecturesare used in applications such as corporate intranets, distributeddatabase applications and video-on-demand services.

Video-on-demand services, for example, typically are designed with anassumption that a user will request an entire movie, and that theselected movie has a substantial length. The video-on-demand servertherefore is designed to support read-only access by several subscribersto the same movie, possibly at different times. Such servers generallydivide data into several segments and distribute the segmentssequentially over several computers or computer disks. This techniquecommonly is called striping, and is described, for example, in U.S. Pat.Nos. 5,473,362, 5,583,868 and 5,610,841. One problem with striping datafor movies over several disks is that failure of one disk or server canresult in the loss of all movies, because every movie has at least onesegment written on every disk.

A common technique for providing reliability in data storage is calledmirroring. A hybrid system using mirroring and sequential striping isshown in U.S. Pat. No. 5,559,764 (Chen et al.). Mirroring involvesmaintaining two copies of each storage unit, i.e., having a primarystorage and secondary backup storage for all data. Both copies also maybe used for load distribution. Using this technique however, a failureof the primary storage causes its entire load to be placed on thesecondary backup storage.

Another problem with sequentially striping data over several disks isthe increased likelihood of what is called a Aconvoy effect. A convoyeffect occurs because requests for data segments from a file tend togroup together at a disk and then cycle from one disk to the next (aAconvoy≅). As a result, one disk may be particularly burdened withrequests at the one time while other disks have a light load. Inaddition, any new requests to a disk must wait for the convoy to beprocessed, thus resulting in increased latency for new requests. Inorder to overcome the convoy effect, data may be striped in a randomfashion, i.e., segments of a data file is stored in a random order amongthe disks rather than sequentially. Such a system is described in ADesign and Performance Tradeoffs in Clustered Video Servers, by R.Tewari, et. al., in Proceedings of Multimedia '96, pp. 144-150. Such asystem still may experience random, extreme loads on one disk, however,due to the generally random nature of data accesses.

None of these systems is individually capable of transferring multiple,independent, high bandwidth streams of data, particularly isochronousmedia data such as video and associated audio data, between multiplestorage units and multiple applications in a scalable and reliablemanner. Such data transfer requirements are particularly difficult insystems supporting capture, authoring and playback of multimedia data.In an authoring system in particular, data typically is accessed insmall fragments, called clips, of larger data files. These clips tend tobe accessed in an arbitrary or random order with respect to how the datais stored, making efficient data transfer difficult to achieve.

SUMMARY OF THE INVENTION

Data is randomly distributed on multiple storage units connected withmultiple applications using a computer network. The data is divided intosegments. Each segment is copied, and each copy is stored on a differentone of the storage units. The selection of each storage unit on which acopy of a segment is stored is random or pseudorandom and may beindependent of the storage units on which other segments of the data arestored. Each segment is stored on at least two of the storage units.

This random distribution of multiple copies of segments of data improvesboth scalability and reliability. For example, when an applicationrequests a selected segment of data, the request may be processed by thestorage unit with the shortest queue of requests so that randomfluctuations in the load applied by multiple applications on multiplestorage units are balanced statistically and more equally over all ofthe storage units. Since the data is processed in terms of its segments,data fragments or clips also are processed as efficiently as all of thedata. The applications may request data transfer from a storage unitonly when that transfer would be efficient and may request storage unitsto preprocess read requests. In certain cases, bandwidth utilization ona computer network is optimized by scheduling data transfers among theclients and storage units. In addition, if one of the storage unitsfails, its load is distributed randomly and nearly uniformly over theremaining storage units. Procedures for recovering from failure of astorage unit also may be provided.

The storage units and applications also may operate independently andwithout central control. For example, each client may use only localinformation to schedule communication with a storage unit. Storage unitsand applications therefore may be added to or removed from the system.As a result, the system is expandable during operation.

This combination of techniques results in a system which can transfermultiple, independent high-bandwidth streams of data between multiplestorage units and multiple applications in a scalable and reliablemanner.

Accordingly, in one aspect, a distributed data storage system includes aplurality of storage units for storing data, wherein copies of segmentsof data stored on the storage units are randomly distributed among theplurality of storage units. Each copy of each segment may be stored on adifferent one of the storage units. Each copy of each segment may beassigned to one of the plurality of storage units according to aprobability distribution defined as a function of relativespecifications of the storage units. The distributed data storage systemmay include a computer-readable medium having computer-readable logicstored thereon and defining a segment table accessible by a computerusing an indication of a segment of data to retrieve indications of thestorage units from the plurality of storage units on which the copies ofthe segment are stored. The plurality of storage units may includefirst, second and third storage units connected to a computer network.

In another aspect, a file system for a computer enables the computer toaccess remote independent storage units over a computer network inresponse to a request, from an application executed on the computer, toread data stored on the storage units. Copies of segments of the dataare randomly distributed among the plurality of storage units. The filesystem is responsive to the request to read data, to select, for eachsegment of the selected data, one of the storage units on which thesegment is stored. Each segment of the requested data is read from theselected storage unit for the segment. The data is provided to theapplication when the data is received from the selected storage units.In this file system, the storage unit may be selected such that a loadof requests on the plurality of storage units is substantially balanced.The storage unit for the segment may be selected according to anestimate of which storage unit for the segment has a shortest estimatedtime for servicing the request.

More particularly, the file system may request data from one of thestorage units, indicating an estimated time. If the first storage unitrejects the request, the file system may request data from another ofthe storage units, indicating another estimated time. The file systemrequests the data from the first storage unit when the second storageunit rejects the request. Each storage unit rejects a request for datawhen the request cannot be serviced by the storage unit within theestimated time. The storage unit accepts a request for data when therequest can be serviced by the storage unit within the estimated time.

The file system may read each segment by scheduling the transfer of thedata from the selected storage unit such that the storage unitefficiently transfers data. More particularly, the file system mayrequest transfer of the data from the selected storage unit, indicatinga waiting time. The data may be requested from another storage unit whenthe selected storage unit rejects the request to transfer the data, orthe file system may request the data from the same storage unit at alater time. Each storage unit rejects a request to transfer data whenthe data is not available to be transferred from the storage unit withinthe indicated waiting time. The storage unit transfers the data when theselected storage unit is able to transfer the data within the indicatedwaiting time.

In another aspect, a file system for a computer enables the computer toaccess remote independent storage units over a computer network inresponse to a request, from an application executed on the computer, tostore data on the storage units. The file system is responsive to therequest to store the data to divide the data into a plurality ofsegments. Copies of each segment are randomly distributed among theplurality of storage units. The file system confirms to the applicationwhether the data is stored.

In this file system, the random distribution of data may be accomplishedby selecting, for each segment, at least two of the storage units atrandom and independent of the storage units selected for other segments.The selected storage units may be requested to store the data for eachsegment. The file system may select a subset of the storage units, andmay selecting the storage units for storing the segment from among thestorage units in the selected subset.

The functionality of the file system also may be provided by anotherapplication or through a code library accessible through an applicationprogramming interface. Accordingly, another aspect is the client or theprocess implemented thereby to perform read or write functions,including selection of a storage unit and scheduling of networktransfer. Another aspect is the storage units or the process implementedthereby to perform read or write functions, including selection of astorage unit and scheduling of network transfer. Another aspect is adistributed computer system implementing such functionality. Theseoperations may be performed by a client or a storage unit using onlylocal information so as to enable a system to be readily expandable.

In another aspect, data is recovered in a distributed data storagesystem having a plurality of storage units for storing the data, whereincopies of segments of the data stored on the storage units are randomlydistributed among the plurality of storage units, when failure of one ofthe storage units is detected. To recover the data, segments of whichcopies were stored on the failed storage unit are identified. Thestorage units on which another copy of the identified segments wasstored are identified. A copy of the identified copies is then randomlydistributed among the plurality of storage units. Such data recovery maybe used in combination with the read and write functionality of a filesystem or distributed storage system described herein.

In another aspect, streams of video data are combined to producecomposited video data which is stored in a distributed system comprisinga plurality of storage units for storing video data, wherein copies ofsegments of the video data stored on the storage units are randomlydistributed among the plurality of storage units. The streams of videodata are read from the plurality of storage units. These streams ofvideo data are combined to produce the composited video data. Thecomposited video data is divided into segments. Copies of the segmentsof the composited video data are randomly distributed among theplurality of storage units. The reading and storage of data may beperformed using the techniques described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings,

FIG. 1 is a block diagram of an example computer system with which thepresent invention may be used;

FIG. 2 illustrates a data structure mapping segments of data to storageunits 42 in FIG. 1;

FIG. 3 is a flowchart describing how data may be captured anddistributed among several storage units in one embodiment of theinvention;

FIG. 4 is a flowchart describing how storage units may process requestsfor storing data in one embodiment of the invention;

FIG. 5 is a flowchart describing how fault recovery may be performedwhen a storage unit becomes unavailable;

FIG. 6 is a flowchart describing how an additional copy of data may bemade;

FIG. 7 is a flowchart describing how a copy of data may be deleted;

FIG. 8 is a flowchart describing how a storage unit may be removed fromthe system;

FIG. 9 is a flowchart describing how data may be archived or copied as abackup;

FIG. 10 is state diagram of a process on a storage unit for notifying acatalog manager of availability of the storage unit;

FIG. 11 illustrates a list of storage units that may be maintained by acatalog manager;

FIG. 12 is a state diagram illustrating how the catalog manager maymonitor a storage unit;

FIG. 13 illustrates a table for tracking equivalency of media datafiles;

FIG. 14 illustrates a list structure for representing a motion videosequence of several clips;

FIG. 15 illustrates a structure of buffer memories for supportingplayback of two streams of motion video data and four streams ofassociated audio data at a client;

FIG. 16 is a flowchart describing how a client may process a multimediacomposition into requests for data from a selected storage unit;

FIG. 17 is a flowchart describing how a client requests a storage unitto transfer data from primary storage into a buffer in one embodiment;

FIG. 18 is a flowchart describing how a storage unit replies to requestsfrom a client in FIG. 17;

FIG. 19 illustrates example disk queues, for prioritizing requests fordisk access to data, and network queues, for prioritizing requests fornetwork transfers of data;

FIG. 20 is a flowchart describing how a client requests a storage unitto transfer data over the network in one embodiment;

FIG. 21 is a flowchart describing how a storage unit processes requeststo transfer data from multiple clients in one embodiment;

FIG. 22 is a flow chart describing an embodiment of a network schedulingprocess performed by a client for transferring data from the client to astorage unit; and

FIG. 23 is a flow chart describing an embodiment of a network schedulingprocess performed by a storage unit for transferring data from a clientto the storage unit.

DETAILED DESCRIPTION

In the following detailed description, which should be read inconjunction with the attached drawings, example embodiments of theinvention are set forth. All references cited herein are herebyexpressly incorporated by reference.

Several problems arise in the design of a scalable and reliabledistributed system that supports transfer of data, particularlymultiple, independent streams of high-bandwidth, time-sensitive datasuch as motion video and associated audio and other temporallycontinuous media, between multiple applications and multiple storageunits. In such a system, an application, for example that is used toauthor a motion video program, may access randomly several smallportions of several different files that may be distributed over severalstorage units. Several applications may require immediate andsimultaneous access to the same data, and any application should be ableto access any piece of media at any time. In a system that is used forbroadcasting or other time sensitive playback, fault tolerance also isdesirable. Finally, the system should be both expandable and scalable ina manner that simplifies the addition of new storage units and newapplications even while the system is in operation. Other desirablecharacteristics of such a system include a long mean time to failure, nosingle point of failure, the capability of being repaired rapidly andwhile operating, tolerance to storage unit failure without disruptingoperation, and the capability of recovering lost data.

In one embodiment, the system includes multiple applications connectedby a computer network to multiple separate and independent storage unitsfor storing data. The data is divided into segments. Each segment iscopied and each copy is stored on a different one of the storage units.The selection of each storage unit is random or pseudorandom and may beindependent of the storage units selected for other segments, such asthe immediately preceding segment. The replication and randomdistribution of data both increases the ability of the system toefficiently transfer data in both directions between applications andstorage and improves fault tolerance. By further controlling whichstorage unit is accessed by a particular application, such as byselecting the storage unit with the shortest queue of requests, randomfluctuations in load are distributed approximately evenly over all ofthe storage units. Applications also may request data transfer with astorage unit only when the transfer would be efficient. In addition, byscheduling communication over the network appropriately, networkcongestion may be reduced and network bandwidth may be used moreefficiently. Central control points may be eliminated by having eachclient use local information to schedule communication with a storageunit.

FIG. 1 illustrates an example computer system 40 in which the presentinvention may be used. The computer system includes a plurality ofstorage units 42. A storage unit is a device with a nonvolatilecomputer-readable medium, such as a disk, on which data may be stored.The storage unit also has faster, typically volatile, memory into whichdata is read from the medium. Each storage unit also has its ownindependent controller which responds to requests for access, includingbut not limited to read and write access, to data stored on the medium.For example, the storage unit 42 may be a server computer which storesdata in a data file in the file system of the server. There may be anarbitrary number of storage units in the computer system 40.

Applications 44 are systems that request access to the storage units 42via requests to the storage units over a computer network 46. Thestorage units 42 may deliver data to or receive data from theapplications 44 over the computer network 46. Applications 44 mayinclude systems which capture data received from a digital or analogsource for storing the data on the storage units 42. Applications 44also may include systems which read data from the storage units, such assystems for authoring, processing or playback of multimedia programs.Other applications 44 may perform a variety of fault recovery tasks.Applications 44 also may be called Aclients;≅ One or more catalogmanagers 49 also may be used. A catalog manager is a database,accessible by the applications 44, that maintains information about thedata available on the storage units 42. This embodiment of the presentinvention may be used to implement a broadcast news system such as shownin PCT Publication WO97/39411, dated Oct. 23, 1997.

In the present invention, data to be stored on the storage units 42 isdivided into segments. Each segment is copied. Each copy is stored on adifferent one of the storage units 42. The selection of the storageunits on which the copies of a segment are stored is random orpseudorandom and may be independent of the storage units on which othersegments of the data are stored. In one embodiment, two consecutivesegments are not stored on the same storage unit. As a result, eachsegment is stored on at least two of the storage units 42. Theprobability distribution for selecting a storage unit for storing aparticular copy of a segment may be uniform over all of the storageunits where the specifications, such as capacity, bandwidth and latency,of the storage units are similar. This probability distribution also maybe a function of the specifications of each storage unit. This randomdistribution of multiple copies of segments of data improves bothscalability and reliability.

An example of this random distribution of copies of segments of data isshown in FIG. 1. In FIG. 1, four storage units 42, labeled w, x, y andz, store data which is divided into four segments labeled 1, 2, 3 and 4.An example random distribution of the segments and their copies isshown, where: segments 1 and 3 are stored on storage unit w; segments 3and 2 are stored on storage unit x; segments 4 and 1 are stored onstorage unit y; and segments 2 and 4 are stored on storage unit z.

The random distribution of segments may be represented in and tracked bya segment table 90A, or catalog, such as shown in FIG. 2. In particular,for data captured from a given source or for data from a given file,each segment, represented by a row 92A, has two copies, called A and B,which are represented by columns 94A. The columns 94A in the segmenttable 90A may be referred herein to as the “A list” or “B list”respectively. Each list alternatively may be represented by a seednumber for a pseudorandom number generator that is used to generate thelist, or by a list or other suitable data structure such as a record,linked list, array, tree, table, etc. When using a pseudorandom numbergenerator, care should be taken to ensure that the storage unitsindicated by the numbers for any given segment in the A and B lists arenot the same. The contents of columns 94 indicate the storage unit onwhich a copy of a segment is stored. Each segment table, or file map,can be stored separately from other segment tables. Segment tables maybe stored together, as a catalog. Catalogs may be stored on a catalogmanager 49, at individual clients, at a central database, or may bedistributed among several databases or clients. Separate catalogs couldbe maintained, for example, for different types of media programs. Forexample, a broadcast news organization may have separate catalogs forsports news, weather, headline news, etc. The catalogs also may bestored on the storage units in the same manner as other data, in whichcase each client could use a seed for a random number generator toaccess the catalog. Such catalogs may be identified by other clients toaccess data or to handle recovery requests, for example, by sending anetwork broadcast message to all catalog managers or clients to obtain acopy of the catalog or of an individual segment table.

In order to access the segments of data, each segment should have aunique identifier. The copies of the segments may have the same uniqueidentifier. The unique identifier for a segment is a combination of aunique identifier for the source, such as a file, and a segment number.The unique identifier for the source or file may be determined, forexample, by a system time or other unique identifier determined whendata is captured from the source or at the time of creation of the file.A file system, as described below, may access the catalog manager toobtain the segment table for each source or file which lists the segmentidentifiers and the storage units on which the copies of the segmentsare stored. Each storage unit also may have a separate file system whichcontains a directory of the segment identifiers and the location on thatstorage unit where they are stored. Application programs executed by aclient may use the identifiers of a source or file, and possibly a rangeof bytes within the source or file to request data from the file systemof the client. The file system of the client then may locate the segmenttable for the source or file, determine which segments need to beaccessed and select a storage unit from which the data should be readfor each segment, using the unique segment identifiers.

Referring again to FIG. 1, when an application 44 requests access to aselected segment of data on one of the storage units 42, the storageunit places the request on a queue 48 that is maintained for the storageunit. Applications may make such requests independently of each other orany centralized control, which makes the system more readily scalable.The selection of a storage unit to which a request is sent may becontrolled such that random fluctuations in the load applied by multipleapplications 44 on multiple storage units 42 are balanced statisticallyand more equally over all of the storage units 42. For example, eachrequest from an application 44 may be processed by the storage unit thathas the shortest queue of requests. In addition, the transfer of databetween applications and storage units may be scheduled to reducenetwork congestion. The requests for data may be performed in two steps:a pre-read request which transfers the data from disk to a buffer on thestorage unit, and a network transfer request which transfers data overthe network from the buffer to the application. To process these twodifferent requests, the queue 48 may include a disk queue and a networkqueue.

This combination of randomly distributed copies of data, the selectionof a storage unit for read access based on the relative loads of thestorage units, and the scheduling of data transfer over the networkprovides a system which can transfer multiple, independenthigh-bandwidth streams of data in both directions between multiplestorage units and multiple applications in a scalable and reliablemanner.

Referring now to FIG. 3, an example process for storing multiple copiesof segments of data in a randomly distributed manner over the severalstorage units will now be described in more detail. The followingdescription is based on the real-time capture of motion video data. Theexample may be generalized to other forms of data, including, but notlimited to other temporally continuous media, such as audio, or discretemedia such as still images or text, or even other data such as sensorydata.

It is generally well-known how to capture real-time motion videoinformation into a computer data file, such as described in U.S. Pat.Nos. 5,640,601 and 5,577,190. This procedure may be modified to includesteps for dividing the captured data into segments, and copying andrandomly distributing the copies of the segments among the storageunits. First, in step 120, the capturing system creates a segment table90 (FIG. 2). An image index, that maps each image to an offset into thestream of data to be captured, also typically is created. The indexedimages may correspond to, for example, fields or frames. The index mayrefer to other sample boundaries, such as a period of time, for otherkinds of data, such as audio. The capturing system also obtains a listof available storage units. One way to identify which storage units areavailable is described in more detail below in connection with FIGS.10-12.

A segment of the data is created by the capturing system in step 121.The size of the segment may be, for example, one quarter, one half orone megabyte for motion video information. Audio information may bedivided into, for example, segments having a size such as one-quartermegabyte. In order to provide alignment, if possible, of the segmentsize to divisions of storage and transmission, the size of the segmentmay be related, i.e., an integer multiple of, to an uncompressed orfixed data rate, disk block and track size, memory buffer size, andnetwork packet (e.g., 64K) and/or cell sizes (e.g., 53 bytes for ATM).If the data is uncompressed or is compressed using fixed-ratecompression, the segment may be divided at temporal sample boundarieswhich provides alignment between the image index and the segment table.Generally speaking, the segment size should be driven to be larger inorder to reduce system overhead, which is increased by smaller segments.On the other hand, there is an increased probability that a convoyeffect could occur if the amount of data to be stored and segment sizeare such that the data is not distributed over all of the storage units.Additionally, there is an increased latency to complete both diskrequests and network requests when the segment sizes are larger.

Next, at least two of the storage units 42 are selected, in step 122, bythe capturing system from the list of storage units available forstoring the selected segment. Selection of the storage units for thecopies of one segment is random or pseudorandom. This selection may beindependent of the selection made for a previous or subsequent segment.The set of storage units from which the selection is made also may be asubset of all of the available storage units. The selection of a set ofstorage units may be random or pseudorandom for each source or file. Thesize of this subset should be such that each storage unit has at leasttwo different segments of the data in order to minimize the likelihoodof occurrence of a convoy effect. More particularly, the data should beat least twice as long (in segments) as the number of storage units inthe set. In addition, the size of the subset should be limited to reducethe probability that two or more storage units in the subset fail, i.e.,a double fault may occur, at any given time. For example, theprobability that two storage units out of five could fail is less thanthe probability that two storage units out of one hundred could fail, sothe number of storage units over which data is distributed should belimited. However, there is a trade off between performance and subsetsize. For example, using randomly selected subsets of ten out ofone-hundred storage units, when two of the one-hundred storage unitsfail, then ten percent of the files are adversely affected. Withoutsubsets, one hundred percent of the files typically would be adverselyaffected.

In the rare likelihood of a double fault, i.e., where two or morestorage units fail, a segment of data may be lost. In a standard videostream, the loss of a segment might result in a loss of one or twoframes in every minute of program material. The frequency of such afault for a given source or file is a function of its bandwidth and thenumber of storage units. In particular, where:

-   -   s=size of lost data in megabytes (MB),    -   n=initial number of storage units,    -   b=average bandwidth of storage units in MB per second,    -   MTBF=mean time between failures,    -   MTTR=mean time to repair or replace,    -   MTDF=mean time for a double fault failure, and    -   SMTBF=total system mean time between failures,

${{SMTBF} = \frac{MTBF}{n}},{and}$${MTDF} = {\frac{1}{MTTR}*\frac{MTBF}{n}*{\frac{MTBF}{\left( {n - 1} \right)}.}}$

As an example, in a system with 100 storage units, each with a capacityof 50 gigabytes, where MTTR is one hour and MTBF is 1000 hours or sixweeks, there likely will be 115 years to double fault failure. If theMTTR is increased to twenty-four hours, then there likely will be 4.8years to double fault failure.

After two storage units are selected, the current segment then is sentto each of the selected storage units in step 124 for storage. Thesewrite requests may be asynchronous rather than sequential. The capturesystem then may wait for all storage units to acknowledge completion ofthe storage of the segment in the step 126. When data must be stored inreal time while being captured, the data transfer in step 124 may occurin two steps, similar to read operations discussed in more detail below.In particular, the client first may request a storage unit to prepare afree buffer for storing the data. The storage unit may reply with anestimated time for availability of the buffer. When that estimated timeis reached, the capture system can request the storage unit to receivethe data. The storage unit then can receive the data in its buffer, thentransfer the data in its buffer to its storage medium and send anacknowledgment to the capture system.

If a time out occurs before an acknowledgment is received by thecapturing system, the segment may be sent again either to the samestorage unit or to a different storage unit. Other errors also may behandled by the capturing system. The operations which ensure successfulstorage of the data on the selected units may be performed by a separatethread for each copy of the segment.

After the data is successfully stored on the storage units, the segmenttable 90 is updated by the capturing system in step 127. If capture iscomplete, as determined in step 128, then the process terminates;otherwise, the process is repeated for the next segment by returning tostep 121. The segment table may be maintained, e.g., in main memory, atthe capture system as part of the file system. While the capturingsystem manages the segment table and selection of storage units in thisexample, other parts of the system could coordinate these activities aswell, such as the catalog manager 49. The updated segment table may besent to, for example, the catalog manager in step 129. Alternatively,the catalog manager may produce the segment table by using accumulatedknowledge of system operation, and may send this table to the capturesystem on request.

FIG. 4 is a flowchart describing in more detail how a storage unitstores a segment of the captured data. The storage unit receives thesegment of data from a capturing system in step 140 and stores the datain a buffer at the storage unit. Assuming the storage unit uses datafiles for storage, the storage unit opens a data file in step 142 andstores the data in the data file in step 144. The catalog manager mayspecify the location where the segment should be stored. The data may beappended to an existing data file or may be stored in a separate datafile. As discussed above, the storage unit or the catalog manager maykeep track of segments by using a unique identifier for each segment andby storing a table mapping the segment identifier to its location on thestorage unit, in step 145. This table may implement the data fileabstraction on the storage unit. When the storage unit actually writesdata to its main storage may depend on other read and write requestspending for other applications. The management of these concurrentrequests is addressed in more detail below. The file then may be closedin step 146. An acknowledgment may be sent to the capturing system instep 148.

When the process of FIGS. 3 and 4 is complete, the captured data israndomly distributed, with at least two copies for each segment, overseveral storage units. Multiple applications may request access to thisdata. The manner in which this access occurs is likely to be random.Accordingly, it should be apparent that any storage unit may receivemultiple requests for both reading data from and writing data to filesstored on the storage unit from multiple applications. In order tomanage the requests, a queue 48 of requests is maintained by each of thestorage units 42, as mentioned above. In the following description of anexample embodiment of the invention, a storage unit maintains twoqueues: one for requests for disk access, and another for requests fornetwork transfers. One embodiment of these disk and network queues isdescribed in more detail below in connection with FIG. 19.

When data is requested by an application program executed on a client44, a storage unit is selected to satisfy the request, since eachsegment of data is stored on at least two storage units. The segmenttable 90A for the requested data is used for this purpose. The selectionof a storage unit may be performed by the application program requestingthe data, by a file system of the client executing the applicationprogram, through coordination among storage units or by anotherapplication such as a catalog manager. The selection may be random orpseudorandom, or based on a least recently used algorithm, or based onthe relative lengths of the queues of the storage units. By selecting astorage unit based on the relative lengths of the queues on theavailable storage units, the load of the multiple applications may bedistributed more equally over the set of storage units. Such selectionwill be described in more detail below in connection with FIG. 16-18.

More details of a particular embodiment of the invention will now bedescribed. For this purpose, the storage unit 42 may be implemented as aserver or as an independently controlled disk storage unit, whereas theapplications 44 are called clients. Clients may execute applicationprograms that perform various tasks. A suitable computer system toimplement either the servers or clients typically includes a main unitthat generally includes a processor connected to a memory system via aninterconnection mechanism, such as a bus or switch. Both the server andclient also have a network interface to connect them to a computernetwork. The network interface may be redundant to support faulttolerance. The client also may have an output device, such as a display,and an input device, such as a keyboard. Both the input device and theoutput device may be connected to the processor and memory system viathe interconnection mechanism.

It should be understood that one or more output devices may be connectedto the client system. Example output devices include a cathode ray tube(CRT) display, liquid crystal displays (LCD), printers, communicationdevices such as a modem or network interface, and video and audiooutput. It should also be understood that one or more input devices maybe connected to the client system. Example input devices include akeyboard, keypad, trackball, mouse, pen and tablet, communicationdevices such as a modem or network interface, video and audio digitizersand scanner. It should be understood the invention is not limited to theparticular input or output devices used in combination with the computersystem or to those described herein.

The computer system may be a general purpose computer system which isprogrammable using a high level computer programming language, such asthe “C” and “C++” programming languages. The computer system also may bespecially programmed, special purpose hardware. In a general purposecomputer system, the processor is typically a commercially availableprocessor, of which the series x86 processors such as the Pentium IIprocessor with MMX technology, available from Intel and similar devicesavailable from AMD and Cyrix, the 680X0 series microprocessors availablefrom Motorola, the Alpha series microprocessor available from DigitalEquipment Corporation, and the PowerPC processors available from IBM areexamples. Many other processors are available. Such a microprocessor mayexecute a program called an operating system, of which the WindowsNT,Windows 95, UNIX, IRIX, Solaris, DOS, VMS, VxWorks, OS/Warp, Mac OSSystem 7 and OS8 operating systems are examples. The operating systemcontrols the execution of other computer programs and providesscheduling, debugging, input/output control, compilation, storageassignment, data management and memory management, and communicationcontrol and related services. The processor and operating system definea computer platform for which application programs in high-levelprogramming languages are written.

Each server may be implemented using an inexpensive computer with asubstantial amount of main memory, e.g., much more than thirty-twomegabytes, and disk capacity, e.g., several gigabytes. The disk may beone or more simple disks or redundant arrays of independent disks (RAID)or a combination thereof. For example, the server may be a Pentium or486 microprocessor-based system, with an operating system such asWindowsNT or a real-time operating system such as VxWorks. The authoringsystem, capturing system and playback system may be implemented usingplatforms that currently are used in the art for those kinds ofproducts. For example, the MEDIACOMPOSER authoring system from AvidTechnology, Inc., of Tewksbury, Mass., uses a Power Macintosh computerfrom Apple Computer, Inc., that has a PowerPC microprocessor and a MacOSSystem 7 operating system. A system based on a Pentium II processor withMMX technology from Intel, with the WindowsNT operating system, also maybe used. Example playback systems include the ASPACE≅ system from PlutoTechnologies International Inc., of Boulder, Colo., or the AIRPLAYsystem from Avid Technology which uses a Macintosh platform. The catalogmanager may be implemented using any platform that supports a suitabledatabase system such as the Informix database. Similarly, an assetmanager that tracks the kinds of data available in the system may beimplemented using such a database.

The memory system in the computer typically includes a computer readableand writeable nonvolatile recording medium, of which a magnetic disk,optical disk, a flash memory and tape are examples. The disk may beremovable, such as a floppy disk or CD-ROM, or fixed, such as a harddrive. A disk has a number of tracks in which signals are stored,typically in binary form, i.e., a form interpreted as a sequence of onesand zeros. Such signals may define an application program to be executedby the microprocessor, or information stored on the disk to be processedby the application program. Typically, in operation, the processorcauses data to be read from the nonvolatile recording medium into anintegrated circuit memory element, which is typically a volatile, randomaccess memory such as a dynamic random access memory (DRAM) or staticmemory (SRAM). The integrated circuit memory element allows for fasteraccess to the information by the processor than does the disk. Theprocessor generally manipulates the data within the integrated circuitmemory and then copies the data to the disk when processing iscompleted. A variety of mechanisms are known for managing data movementbetween the disk and the integrated circuit memory element, and theinvention is not limited thereto. It should also be understood that theinvention is not limited to a particular memory system.

It should be understood the invention is not limited to a particularcomputer platform, particular processor, or particular high-levelprogramming language. Additionally, the computer system may be amultiprocessor computer system or may include multiple computersconnected over a computer network.

As stated above, each storage unit 42, if accessed through server, andeach application 44 may have a file system, typically part of theoperating system, which maintains files of data. A file is a namedlogical construct which is defined and implemented by the file system tomap the name and a sequence of logical records of data to locations onphysical storage media. While the file system masks the physicallocations of data from the application program, a file system generallyattempts to store data of one file in contiguous blocks on the physicalstorage media. A file may specifically support various record types ormay leave them undefined to be interpreted or controlled by applicationprograms. A file is referred to by its name or other identifier byapplication programs and is accessed through the file system usingcommands defined by the operating system. An operating system providesbasic file operations for creating a file, opening a file, writing afile, reading a file and closing a file. These operations may besynchronous or asynchronous, depending on the file system.

In the present invention, the data of a file or source is stored insegments, of which copies are randomly distributed among multiplestorage units.

Generally speaking for most file systems, in order to create a file, theoperating system first identifies space in the storage which iscontrolled by the file system. An entry for the new file is then made ina catalog which includes entries indicating the names of the availablefiles and their locations in the file system. Creation of a file mayinclude allocating certain available space to the file. In oneembodiment of the invention, a segment table for the file may becreated. Opening a file typically returns a handle to the applicationprogram which it uses to access the file. Closing a file invalidates thehandle. The file system may use the handle to identify the segment tablefor a file.

In order to write data to a file, an application program issues acommand to the operating system which specifies both an indicator of thefile, such as a file name, handle or other descriptor, and theinformation to be written to the file. Generally speaking, given theindicator of the file, an operating system searches the directory tofind the location of the file. The data may be written to a knownlocation within the file or at the end of the file. The directory entrymay store a pointer, called a write pointer, to the current end of thefile. Using this pointer, the physical location of the next availableblock of storage may be computed and the information may be written tothat block. The write pointer may be updated in the directory toindicate the new end of the file. In one embodiment of the invention,the write operation randomly distributes copies of segments of the fileamong the storage units and updates the segment table for the file.

In order to read data from a file, an application program issues acommand to the operating system specifying the indicator of the file andmemory locations assigned to the application where the read data shouldbe placed. Generally speaking, an operating system searches itsdirectory for the associated entry given the indicator of the file. Theapplication program may specify some offset from the beginning of thefile to be used, or, in a sequential file system, the directory mayprovide a pointer to a next block of data to be read. In one embodimentof the invention, the selection of a storage unit and the scheduling ofdata transfer is implemented as part of the read operation of the filesystem of the client.

The client may use a file system or a special code library with adefined application programming interface (API) to translate requestsfor portions of a file into requests for segments of data from selectedstorage units. The storage unit may have its own file system which maybe entirely separate from the client file system. All of the segments ona storage unit may be stored, for example, in a single file at thestorage unit. Alternatively, the client file system may use the storageunits over the network as raw storage, using the catalog manager andsegment tables to implement the file abstraction. In such a case, thesegment table for a file also may indicate the locations of each segmenton the storage units selected for the segment.

A primary advantage of using a file system is that, for an applicationprogram, the file is a logical construct which can be created, opened,written to, read from and closed without any concern for the physicalstorage medium or location on that medium used by the operating systemto store the data. In a network file system, the file system managesrequests for data from a specified file from the various storage units,without requiring an application program to know any details about thephysical storage where the data is stored or the computer network. Ifthe storage unit has its own independent file system, the client filesystem also need not know details of the storage mechanism of thestorage units. In the present invention, the storage units may use, forexample, the file system associated with their own operating system,such as the WindowsNT file system or the file system of a real timeoperating system such as VxWorks, or a file system that allowsasynchronous operations. As stated above, the storage units areinterconnected with the clients and, optionally, the catalog managerusing a computer network. A computer network is a set of communicationschannels interconnecting a set of computer devices or nodes that cancommunicate with each other. The nodes may be computers such as theclients, storage units and catalog managers, or communication devices ofvarious kinds, such as switches, routers, gateways and other networkdevices. The communication channels may use a variety of transmissionmedia including optical fibers, coaxial cable, twisted copper pairs,satellite links, digital microwave radio, etc.

A computer network has a topology which is the geometrical arrangementof the connection of the nodes by the network. Kinds of topologiesinclude point-to-point connection, linear bus, ring connection, starconnection, and multiconnected networks. A network may use variouscombinations of these basic topologies. In the present invention, thetopology will vary depending on the physical installation. Anon-blocking, switch-based network in which each node, i.e., client orstorage unit, is connected directly to the same switch may be used. Insome implementations, multiple clients and storage units may beconnected on a physical loop or subnetwork which are interconnected intoa switching fabric. The system also may be connected using multipleswitches.

The network also has a network architecture which defines the protocols,message formats, and other standards to which communication hardware andsoftware must conform in order for communication to occur betweendevices on the network. A commonly-used network architecture is theInternational Standards Organization seven-layer model known as the OpenSystems Interconnection reference model. The seven layers are theapplication, presentation, session, transport, network, link andphysical layers. Each machine communicates with any other machine usingthe same communication protocol at one of these layers.

In one embodiment, the link layer preferably is one that retains theorder of packets as they are received at the client in order to avoidthe potential for an unlimited latency. Accordingly, suitable link layerprotocols include asynchronous transfer mode (ATM) networks, such asOC3, OC12, or higher bandwidth networks. An ATM system operating in theAAL5 mode is preferable. Ethernet networks with 100 Tx to gigabit (1,000Tx) capacity also may provide efficient packet transmission from thesource to the destination. Suitable Ethernet network platforms areavailable, for example, from 3Com of Santa Clara, Calif. An example ATMsystem is available from Fore Systems of Warrendale, Pa. or Giga-Net, ofConcord, Mass. A FibreChannel, FDDI or HIPPI network also may be used.The different clients, the catalog manager and the storage units all maycommunicate using the link layer protocol. Communication at this layeralso reduces overhead due to memory copies performed to processencapsulated data for each layer's protocol. A bandwidth distributednetwork file system from Polybus Systems Corporation in Tyngsboro,Mass., may be used.

Having now described computer platforms for an embodiment of a system inaccordance with the invention, some additional operations and details ofone embodiment will now be described.

In one embodiment of this system, there are processes for maintainingthe storage units and the data stored on the storage units. For example,fault recovery procedures may involve the creation of additional copiesof a file. Additionally, files may be deleted or added based on the needfor availability of, i.e., reliability of access to, the file. Finally,some maintenance procedures may involve deleting files on a storageunit, copying the files to another storage unit and removing the storageunit from the system. A file also may be archived, or removed from thesystem to archival storage. These processes will now be described inmore detail in connection with FIGS. 5-9. Such data management processesmay be performed by the catalog manager, another storage unit, or aclient. The performance of these processes by a client would not occupythe resources of the catalog manager or storage units, which may be usedfor other more important tasks, such as replying to client requests fordata.

FIG. 5 is a flowchart describing in more detail how fault recovery maybe performed when a storage unit becomes unavailable after its failureis detected. One way to detect such failure is described in more detailbelow in connection with FIGS. 10-12. Repeated failures to respond torequests also may be used to indicate failures. The success of thisprocess depends on the number of copies of each segment within thesystem. Given a number N of copies, then N−1 storage units may fail andthe system still will operate without loss of data. After a storage unitfails, a new storage unit may be installed in its place, with lost datarestored, or the lost data may be recreated and distributed over theremaining storage units.

Additional copies of data may be made by first selecting the data, e.g.,a file or source to be recovered, in step 200. The file to be recoveredmay be selected by a priority ordering, and may be selected eitherautomatically or manually. This kind of recovery allows data from somefiles to be reconstructed and made available before data from otherfiles is recovered. The lost segments of the data, i.e., those stored onthe lost storage unit, are identified in step 202 using the segmenttable for the source. A new storage unit for each lost segment isselected in step 204, typically in the same manner as when data isoriginally captured, when a new storage unit is not available to replacethe failed storage unit. Alternatively, the replacement storage unit isselected. A copy of the lost segment is read from an alternate storageunit in step 206 and stored in the selected storage unit. The fileoperations for steps 204 through 208 may be asynchronous and performedby separate threads for each segment. Such operation takes advantage ofthe many-to-many read/write capability provided in this networkarchitecture. The segment table for the file then is updated upon thesuccessful completion of the copy operation in step 208. When theprocess is complete, the catalog manager may be updated with the newsegment table in step 209, if a catalog manager maintains the segmenttables. If the original segment table was represented by a seed to apseudorandom sequence generator, the actual table may need to be createdand modified.

The speed of repopulation and redundancy restoration for an unloadedsystem using this

$\frac{s}{\left( {n - 1 + d} \right)\left( {b/2} \right)},$

process is defined by the following equation:where:

-   -   s=size of lost files in megabytes (MB),    -   n=initial number of storage units,    -   b=average bandwidth of storage units, expressed in MB/second,        and    -   d=user demand load, expressed in MB/second.        For example, if access to 50 gigabytes of storage is lost        because one of ten storage units fails, then with n=10 storage        units, with unit bandwidth b=10 MB/sec., then (n−1)=9 and        (b/2)=5. Thus, recovery would take approximately 20 minutes with        no other loads. This absolute recovery speed generally is        reduced as a reciprocal of the varying playback load to clients,        e.g., a 50% load results in 200% increase in repopulation time.        When invoked, the redistribution task can run at a very fast        rate with multiple storage unit checkerboard switched to        multiple storage units, but repopulation activities operate        opportunistically, subordinated to client file service requests.        The net effect is only a slight loss of total bandwidth of the        storage units due to the failed storage unit. Prioritization of        the file selection for recovery ensures that the most important        files are recovered most quickly.

FIG. 6 is a flowchart describing in more detail how an additional copyof data may be made. This process may be invoked to make additional datacopies available of mission critical or high-demand data. A date-stampmay be given to the new copy to indicate when the copy may be deleted.Given selected data, a segment of the data is selected in step 210. Eachsegment is assigned randomly a new storage unit in step 212, ensuringthat each storage unit has at most one copy of a given segment. Next,the segment is stored on the selected storage unit in step 214. Uponsuccessful completion of the storage of that segment, the segment tablefor the data is updated in step 216. If all of the segments of the datahave not yet been copied, as determined in step 217, the process repeatsby returning to step 210 to select the next segment of the data. Whenthe process is complete, the catalog manager may be updated with the newsegment table in step 218, if the catalog manager maintains the segmenttables. While this process is sequential over the segments, each segmentmay be processed using a separate thread, and the file operation of step214 may be asynchronous. Such processing enables the copy to be madevery quickly. With this procedure, the segment table still may berepresented using the seed for the pseudorandom number generator.

FIG. 7 is a flowchart describing in more detail how a copy of data isdeleted. This process may be invoked, for example, when data is nolonger in high demand. For example, a date stamp on a copy may be usedto indicate when the data should be deleted. Given the segment tableshown in FIG. 2 for given data, one of the sets of copies, i.e., acolumn in the table, is selected in step 220. Each segment in the columnis deleted in step 222. Upon successful completion of the deleteoperation in step 222 for each segment, the segment table is updated instep 224. Steps 222 and 224 are repeated for segment. This process maybe sequential over the segments or each segment may be processed by aseparate thread. When the process is complete, the catalog manager maybe updated with the new segment table in step 226, if the catalogmanager maintains the segments tables.

FIG. 8 is a flowchart describing how an otherwise active storage unitmay be removed from the system. The data available on the storage unitis identified, for example by identifying a list of its files using itsfile system. First, the storage unit is made unavailable for writing newsegments. This step may be accomplished, for example, by notifying thecatalog manager or by sending a broadcast message to all clients. Thesegments of each file are redistributed on the other storage unitsbefore the storage unit is removed from the system. Given this list offiles, the next file to be processed is selected in step 230. Using thesegment table, all segments of this file on the storage unit areidentified in step 232. The next segment to be processed is selected instep 234. The selected segment is assigned a new storage unit in step235 by a random selection from the remaining storage units, assuringthat no storage unit has more than one copy of a given segment. The datais then written to the newly selected storage unit in step 236. Uponsuccessful completion of that write operation, the segment table isupdated. When all the segments for a given file are redistributed, asdetermined in step 238, the segment table may be sent to the catalogmanager if appropriate in step 239. The segments may be processedsequentially or by separate threads using asynchronous file operations.The segments may be deleted from the old storage unit, as determined instep 237, after the catalog manager is updated. Processing continueswith the next file, if any, as determined in step 240. If all files havebeen redistributed, this process is complete and the storage unit may beremoved from the system.

FIG. 9 is a flowchart describing how data may be archived or copied forbackup. This process involves copying of one copy of each segment of thedata from the available storage units into a backup storage system, suchas an archival storage medium. In addition, each copy set may be deletedfrom all storage units. This process may be performed by selecting acopy set, e.g., the A list, from a column of the segment table in step250. Alternatively, each segment may be read in order and the selectionof a storage unit for each segment may be performed using techniquesapplied by other applications as described above. Each segment from theselected copy set is read from its storage unit and is stored on astorage medium in step 252. Upon successful copying of each segment tothe storage medium, all of the remaining segments from all the remainingcopy sets may be deleted from the storage units in step 254. Thesegments may be processed sequentially or by separate threads usingasynchronous file operations. The catalog manager then may be updated instep 256.

How the storage units may be monitored to determine availability and todetect failures will now be described in connection with FIGS. 10through 12. There are several ways to determine whether storage unitsare available, including polling the storage units, handling exceptionsfrom the storage units, or by the storage units periodically informingan application or applications of their availability. In one embodimentof the invention, in addition to maintaining a catalog of segment tablesfor each file, the catalog manager 49 or some other client may monitorwhich storage units 42 are active in the system. One method formonitoring the storage units is shown in FIGS. 10-12. Each storage unitavailable on the system establishes a process which periodically informsthe catalog manager that it is available. In particular, this processmay be considered as a state machine having a first state 60 in whichthe storage unit periodically increments a counter, for example, inresponse to a timer interrupt or event from a system timer. When thiscounter reaches a certain predetermined amount, such as a hundredmilliseconds, a transition to another state 62 occurs. In the transitionto state 62, a signal, called a Aping,≅ is sent to the catalog managerby the storage unit. This signal may be a small message, even one ATMcell, that does not require much bandwidth to transmit. This signal mayinclude an identifier of the storage unit, and possibly otherinformation such as the capacity, efficiency and/or bandwidthavailability of the storage unit. At the next timer interrupt or event,the counter is reset and a transition back to state 60 occurs.

The catalog manager may keep track of the available storage units. Forthis purpose, the catalog manager may use a list 70 of storage units, anexample of which is shown in FIG. 11. This list of storage units may beimplemented as a table indexed by the identifiers of the storage unitsas indicated at 72. If the storage unit is present or available, thebandwidth, memory capacity or other information about the power of thestorage unit is made available in column 74. The count since the lastAping≅ from the storage unit also is present as indicated in column 76.If this count exceeds a predetermined amount, such as three hundredmilliseconds, the storage unit is considered to be out of service andfault recovery procedures, such as described above, may be followed. Anexample tracking process which maintains the list 70 of storage unitswill now be described in more detail in connection with FIG. 12.

FIG. 12 is a state machine describing a tracking process which may beperformed by the catalog manager to determine which storage units areavailable. One of these state machines may be established for eachstorage unit as a process on the catalog manager. The first state 80 isa waiting state in which the count value 76 for the storage unit in thelist 70 of storage units is incremented for the storage unit in responseto periodic timer interrupts. When a “ping” is received from the storageunit, the transition occurs to state 82. In state 82, the presence ofthis storage unit in list 70 is verified. If the storage unit is in thelist 70, the count 76 for the storage unit is reset, other informationabout the storage unit may be updated, and a transition back to state 80occurs. If the storage unit is not in the list, it is added to the listwith a reset count and a transition back to state 80 occurs. After agiven increment, if the count for the storage unit is greater than apredetermined time out value, such as three hundred milliseconds, faultrecovery procedures are performed. In particular, the storage unit isremoved from list 70 and fault tolerant procedures are performed instate 84. If a “ping” from a storage unit is received by the catalogmanager and if that storage unit does not have a corresponding trackingprocess, then the catalog manager adds the storage unit to the list andcreates a tracking process for the storage unit.

In addition to having a catalog manager 49, the system also may includea database, called an asset manager, which stores a variety of dataabout the media sources available in the system such as an index foreach file. The catalog manager and asset manager also may be combined.One useful kind of information for storing in the asset manager is atable, shown in FIG. 13, that relates equivalent data files based on asource identifier and a range within that source, such as shown in U.S.Pat. No. 5,267,351. The source identifier is an indication of theoriginal source of data, which may be an analog source, whereas the dataactually available is a digitized copy of that source stored on thestorage units. In particular, the table has an entry for a sourceidentifier 100, a range within the source identifier 102, and anindication 104, such as list of data files, of equivalent data from thatsource. The list 104 may be used to identify one of the data files for asource, and in turn access the segment table for that file to determinewhere segments of the data are distributed on the various storage units.The segment table 90A of FIG. 2 may be incorporated into this list 104of FIG. 13 as shown at 106 and 108. Such equivalency among data also maybe maintained by any application program.

Since the catalog manager is a database that monitors how data isdistributed on the various storage units, it also should be designed toenhance fault tolerance and availability and to reduce its likelihood ofbeing a bottleneck. Accordingly, the catalog manager should beimplemented using conventional distributed database managementtechniques. Also, highly available machines, such as those from MarathonTechnologies, Tandem Computers, Stratus, and Texas Micro, Inc., may beused to implement the catalog manager. There also may be several catalogmanagers that are used by separate client applications. Alternatively,each client application may maintain its own copy of catalogs locally,using standard techniques to maintain consistency between multiplecopies of the data. In this manner, a catalog manager is not a centralpoint of failure. A client also may act as its own catalog manager. Thecatalogs also may be treated as data of which copies of its segments arerandomly distributed among the storage units. Each client may have asegment table, or random number generator seed representing the segmenttable, for each catalog.

Having now described how data may be captured and stored onto storageunits, and how the storage of data on the storage units may be managed,client applications that perform authoring and playback will now bedescribed in more detail in connection with FIGS. 14 and 15.

There are several kinds of systems that may be used to author, processand display multimedia data. These systems can be used to modify thedata, define different combinations of data, create new data and displaydata to a user. A variety of techniques are known in the art forimplementing these kinds of systems.

Multimedia authoring, processing and playback systems typically have adata structure which represents the multimedia composition. The datastructure ultimately refers to clips of source material, such asdigitized video or audio, using an identifier of the source material,such as a unique identifier or a file name, and possibly a temporalrange within the source material defining the clip. The identifier maybe of a type that can be used with a list of equivalent data files toidentify a file name for the source material. An index may be used totranslate the temporal range in the source into a range of bytes withina corresponding file. This range of bytes may be used with the segmenttable for the file to identify segments of data that are needed and thestorage units from which the data will be retrieved.

FIG. 14 shows an example list structure that may be used to representpart of a multimedia composition. In an example shown in FIG. 14, thereare several clips 260, each of which includes a reference to a sourceidentifier, indicated at 262, and a range within the source, asindicated at 264. Generally, there may be such a list for each track ofmedia in a temporal composition. There are a variety of data structureswhich may be used to represent a composition. In addition to a liststructure, a more complex structure is shown in PCT PublishedApplication WO93/21636 published on Oct. 28, 1993. Other examplerepresentations of multimedia compositions include those defined by OpenMedia Framework Interchange Specification from Avid Technology, Inc.,QuickTime from Apple Computer, DirectShow from Microsoft, and Bento alsofrom Apple Computer, and as shown in PCT Publication WO96/26600.

The data structure described above and used to represent multimediaprograms may use multiple types of data that are synchronized anddisplayed. The most common example is a television program or filmproduction which includes motion video (often two or more streams) withassociated audio (often four or more streams). As shown in FIG. 15, theclient computer may have a corresponding set 290 of memory buffers 294allocated in the main memory. Each buffer may be implemented as aAserializing≅ buffer. In other words, the client inserts data receivedfrom a storage unit into these independently accessible portions andreads from the set of buffers sequentially. Since requests may be sentto several storage units and data may be received at different times forthe same stream, the buffers may not be filled in sequence when written,but are read out in sequence to be displayed. In FIG. 15, the filled inbuffers indicate the presence of data in the buffer. Any empty buffermay be filled at any time as indicated at 293 and 295. However, each setof buffers has a current read location 291 from which data is read andwhich advances as time progress as indicated in 297. A subset 292, 296of these buffers may be allocated to each stream of data.

Each buffer in the set of buffers has a size that corresponds to a fixednumber of segments of data, where the segment size is the size of filesegments stored on the storage units. There may be several, e.g., four,audio buffers per stream 292 of audio data, where each buffer cancontain several, e.g., four, segments. Similarly, each video stream 296may have several, e.g., four, buffers each of which contains several,e.g., four, segments. Each of the buffers may be divided intoindependently accessible portions 298 that correspond in size to thesize of data packets for which transfer is scheduled over the network.

Because the video and audio data may be stored in different data filesand may be combined arbitrarily, better performance may be obtained ifrequests for data for these different streams on the client side aremanaged efficiently. For example, the client application may identify astream for which data can be read, and then may determine an amount ofdata which should be read, if any. A process for performing this kind ofmanagement of read operations is shown in U.S. Pat. No. 5,045,940. Ingeneral, the client determines which stream has the least amount of dataavailable for display. If there is a sufficient amount of buffer spacein the set of buffers for that stream to efficiently read an amount ofdata, then that data is requested. It is generally efficient to readdata when the available space in memory for the selected stream is largeenough to hold one network transmission unit of data. In the presentinvention, when it is determined that data for a stream should berequested, each segment of the data is requested from a storage unitselected from those on which the segment is stored.

A general overview of a process by which a composition may be convertedinto requests for data in order to display the data will now bedescribed in connection with FIG. 16. In order to know what files torequest from the storage unit, an application program executed on theclient system may convert a data structure representing a composition,such as shown in FIG. 14, into file names and ranges within those filesin step 270 in FIG. 16. For example, for each source identifier andrange within that source, a request may be sent to the asset manager. Inresponse, the asset manager may return a file name for a file containingequivalent media corresponding to the received source identifier andrange. The segment table for the file and the list of available storageunits also may be catalog manager.

When the client requests a segment of data for a particular data stream,the client selects a storage unit, in step 272, for the segment that isrequested. This selection, in one embodiment, will be described in moredetail below in connection with FIGS. 17 and 18. In general, the storageunit with the shortest queue 48 (FIG. 1) is selected. The client thenreads the data from the selected storage unit for the segment, in steps274 through 278. Step 274 may be understood as a pre-read step in whichthe client sends a request to a storage unit to read desired data fromnonvolatile storage into faster, typically volatile storage. The requestto the storage unit may include an indication of how much time isrequired from the time the request is made until that requested datamust be received at the client, i.e., a due time. After a pre-readrequest is accepted, the client waits in step 276. The request is placedin the storage unit=s queue 48, and the due time may be used toprioritize requests as described below. Data is transferred from thestorage unit in step 278 after data becomes available in a buffer at thestorage unit. This step may involve scheduling of the network usage totransfer the data to maximize efficiency of network utilization. Thereceived data is stored in the appropriate buffer at the client, andultimately is processed and displayed in step 280.

There are several ways to initiate the pre-read requests, includingselection of a storage unit, in step 274 and the data transfer in step278. For example, the MediaComposer authoring system from AvidTechnology, Inc., of Tewksbury, Mass., allows a user to set either anumber of clips or an amount of time as a look-ahead value, indicatinghow far ahead in a composition the application should initiate readrequests for data. A program schedule for a television broadcastfacility also may be used for this purpose. Such information may be usedto initiate selection of a storage unit and pre-read requests. Suchpre-reads may be performed even if buffer space is not available inbuffers 290 (FIG. 15), as is shown in European patent application0674414A2, published Sep. 9, 1995. The amount of available space in thebuffers 290 (FIG. 15) may be used to initiate data transfers in step 278(FIG. 16), or to initiate both pre-reads (step 274) and data transfers(step 278).

One process which enables a client to make an adequate estimate of whichstorage unit has the shortest queue of requests, without requiring anexhaustive search of all the available storage units, will now bedescribed in connection with FIGS. 17 and 18. First, the client sends arequest with a threshold El to a first storage unit in step 330. Thethreshold El is a value indicating an estimate of time by which therequest should be serviced. This estimate may be expressed as a timevalue, a number of requests in the disk queue of the storage unit, suchas four, or other measure. The meaning of this threshold is that therequest should be accepted by the storage unit if the storage unit canservice the request within the specified time limit, for example. Theclient receives a reply from the storage unit in step 332. The replyindicates whether the request was accepted and placed in the disk queueof the storage unit or whether the request was rejected as determined instep 334. If the request is accepted, the client is given an estimate oftime at which the data will be available in a buffer at the storage unitin step 336. For example, if the data for the requested segment alreadyis in a buffer, the storage unit indicates that the data is immediatelyavailable. The client then may wait until it is time to request transferof the data (step 278 in FIG. 16) some time after the estimated time haspassed. If the request is rejected, an estimate of the amount of timethe storage unit actually is likely to take, such as the actual size innumber of entries of the disk queue, is returned from the storage unit.This actual estimate is added to a value K to obtain a threshold E2 instep 340. The value K may be two, if representing a number of disk queueentries. Threshold E1 and value K may be user-definable. A request issent to a second storage unit in step 342 indicating the threshold E2.The client then receives a reply in step 344, similar to the replyreceived in step 332. If this reply indicates that the request wasaccepted, as determined in 346, the client has an estimate of time atwhich the data will be available at the second storage unit, asindicated in step 336 after which the client may wait to schedule thedata transfer. Otherwise, an unconditional request, one with a verylarge threshold, is sent to the first storage unit in step 348. Anacknowledgment then is received in step 350 indicating the estimate oftime at which the data will be available in a buffer at the storageunit, as indicated at step 336.

The storage unit, on the other hand, does not know whether it is thefirst or second storage unit selected by the client when it receives arequest. Rather, the storage unit simply receives requests as indicatedin step 360. The threshold indicated in the request is compared to thestorage unit=s own estimate of the time the client will need to wait instep 362, for example by comparing the size of the disk queue of thestorage unit to the specified threshold. If the threshold in the requestis greater than the estimate made by storage unit, the request is placedin the disk queue and an estimate of the time when the data will beavailable in a buffer at the storage unit is determined in step 364.This estimate may be determined, for example, based on disk accessspeed, disk queue length and possibly a running average of recentperformance. An acknowledgment is sent to the client in step 366including the estimated time of availability of the data in the bufferat the storage unit. Otherwise, a rejection is sent in step 368indicating this estimate, such as the actual size of the disk queue.

The storage unit may keep track of which segments are in which bufferson the storage unit. Segment data may be read from the storage mediuminto any free buffer or into a buffer occupied by the least recentlyused segment. In this manner, data for a segment may be immediatelyavailable in a buffer if that segment is requested a second time.

As an alternative, a client may use another method to select a storageunit from which data will be retrieved, as discussed below. Aftersending the request, the client may receive an acknowledgment from thestorage unit indicating that the request is in the disk queue at thestorage unit. Instead of receiving an estimate of time at which the datawill be available in a buffer at the storage unit, the client may waituntil a ready signal is received indicating that the storage unit hasread the requested data into a specified buffer memory at the storageunit. During this waiting period, the client may be performing othertasks, such as issuing requests for other data segments, displaying dataor processing data. One problem with this alternative is that the clientaccepts an unsolicited message, i.e., the ready signal from the storageunit, in response to which the client changes context and processes themessage. The client could be busy performing other operations. Althoughthis process does provide a more accurate estimate of the time at whichdata is available in a buffer at the storage unit, the ability to changecontexts and process incoming messages quickly involve more expensivehardware at the client.

There are several other ways a storage unit may be selected from thesegment table for a file. For example, when a client is making a fileread request, the client can pick randomly from either the “A” list or“B” list for the file in question. Alternatively, the client may reviewall of its currently outstanding requests, i.e., requests sent but notyet fulfilled, and pick which storage unit out of the storage units onthe A and B lists for the segment currently has the fewest outstandingrequests. This selection method may reduce the chance of a clientcompeting with its own outstanding requests, and will tend to spreadrequests more evenly over all the storage units. Alternatively, ratherthan examining outstanding requests, a client may examine a history ofits recent requests, e.g., the last “n” requests, and for the nextrequest pick whichever storage unit from the A list and B list for thesegment has been used less historically. This selection method tends tospread requests more evenly over all the storage units, and tends toavoid a concentration of requests at a particular storage unit. Theclient also may request from each storage unit a measure of the lengthof its disk queue. The client may issue the request to the storage unitwith the shortest disk queue. As another possibility, the client maysend requests to two storage units and ultimately receive the data fromonly one. Using this method on a local area network, the client maycancel the unused request. On a wide area network, the storage unit thatis ultimately selected may cancel the unused request at the otherstorage unit.

As stated above, a storage unit will likely receive multiple requestsfrom multiple applications. In order to manage the requests frommultiple applications to ensure that the most critical requests arehandled first, a queue 48 (FIG. 1) is maintained for each storage unit.The queue may be maintained in several parts, depending on thecomplexity of the system. In particular, the storage unit may maintaindifferent queues for disk access and for network transfers. In addition,the queue may segregate requests from time-sensitive applications usingdata having specific due times, e.g., for playback to broadcast, fromrequests from other applications, such as capture systems, authoringtools or service and maintenance applications. Storage requests may beseparated further from requests from authoring tools and requests fromservice and maintenance programs. Requests from authoring tools may beseparated further from service and maintenance requests.

FIG. 19 illustrates one embodiment of queue 48, utilizing a disk queue300 and a network queue 320. The disk queue has four subqueues 302, 304,306 and 308, one for each of the playback, capture, authoring andservice and maintenance client programs, respectively. Similarly, thenetwork queue 320 has four subqueues 322, 324, 326 and 328. Each queueincludes one or more entries 310, each of which comprises a requestfield 312 indicating the client making the request and the requestedoperation, a priority field 314 indicating the priority of the request,and a buffer field 316 indicating the buffer associated with therequest. The indication of the priority of the request may be adeadline, a time stamp, an indication of an amount of memory availableat the client, or an indication of an amount of data currently availableat the client. A priority scheduling mechanism at the storage unit woulddictate the kind of priority stamp to be used.

The priority value may be generated in many ways. The priority value foran authoring or playback system is generally a measure of time by whichthe application must receive the requested data. For example, for a readoperation, the application can report how much data (in milliseconds orframes or bytes) it has available to play before it runs out of data.The priority indication for a capture system is generally a measure oftime by which the client must transfer the data out of its buffers tothe storage unit. For example, for a write operation, the applicationcan report how much empty buffer space (in milliseconds, frames orbytes) it has available to fill before the buffer overflows. Usingmilliseconds as a unit of measure, the system may have an absolute timeclock that could be used as the basis for ordering requests in the queue48. In this case, all applications and storage units may be synchronizedto the absolute time clock. If such synchronization is not practical,the application may use a time that is relative to the application thatindicates how much time from the time the request is made that may passuntil the requested data should be received by the client. Assuming lowcommunication latency, the storage unit may convert this relative timeto an absolute time that is consistent with the storage unit.

The storage unit processes the requests in its disk queues 302-308 intheir priority order, i.e., operating on the requests in the highestpriority queue first, in order by their priority value, then therequests in successively lower priority queues. For each request, thestorage unit transfers data between the disk and the buffer indicated bythe request. For a read request, after the request is processed, therequest is transferred from the disk queue to the network queue. For awrite request, the request is removed from the disk queue after thewrite operation completes successfully.

In an embodiment of the invention to be described in more detail below,the storage unit uses the network queue to prioritize network transfersin the process of scheduling those transfers. In this embodiment,clients request transfer of data over the network. If a storage unitreceives two such requests at about the same time, the storage unitprocesses the request that has a higher priority in its network queue.For a read request, after the request is processed, the request isremoved from the network queue. For a write request, the request istransferred from the network queue to the disk queue, with a prioritydepending on the availability of free buffers, after the transfercompletes successfully. If the time has passed for a request in thenetwork queue to be processed, the request may be dropped indicatingthat the client is no longer operating or did not request the networktransfer in time.

Data transfers between the storage units and clients over the computernetwork may be scheduled to improve efficiency. In particular,scheduling data transfers improves bandwidth utilization of the computernetwork. Such scheduling of the network usage should be performedparticularly if the bandwidth of the link between a client and a switchis on the same order of magnitude as the bandwidth of the link betweenthe storage unit and the switch. In particular, if the storage unitsends data and the client receives data at the link speed of theirrespective network connections, data is not likely to accumulate at anetwork switch or to experience other significant delays.

In order to enforce such utilization of the network, a mechanism may beprovided that forces each client to receive data from only one storageunit, and that forces each storage unit to send data to only one client,at any given time. For example, each client may have only one token. Theclient sends this token to only one storage unit to request transfer ofthe data for a selected segment. The token may indicate the deadline bywhich the data must be received by the client, i.e., the prioritymeasure, and the specified segment. Each storage unit sends data to onlyone client at a time, from which it has received a token. The storageunit only accepts one token at a time. After the data is transferred,the storage unit also returns the token.

Another network scheduling process will now be described in connectionwith FIGS. 20 and 21. This process provides a similar result but doesnot use a token. Rather a client requests a communication channel with astorage unit, specifying a segment and an amount of time E3 that theclient is willing to wait for the transfer to occur. The client also mayspecify a new due time for the segment by which the client must receivethe data.

Referring now to FIG. 20, the client process for transferring data overthe network will now be described. At any point in time during theplayback of a composition, each buffer has a segment of data associatedwith it and a time by which the data must be available in the buffer forcontinuous playback. As is known in the art, the application associateseach of the buffers with a segment during the playback process. As shownabove in connection with FIGS. 17 and 18, each segment that a client haspreread has an associated estimated time by which the data will beavailable at the storage unit. Accordingly, the client may order thebuffers by their due time and whether the requested data is expected tobe available in a buffer at the storage unit. This ordering may be usedby the client to select a next buffer for which data will be transferredin step 500. The client requests a communication channel with thestorage unit in step 502, specifying a waiting time E3. This value E3may be short, e.g., 100 milliseconds, if the client does not need thedata urgently and if the client may perform other operations moreefficiently. This value E3 may be longer if the client needs the dataurgently, for example, so that it does not run out of data for one ofits buffers. In step 504, the client receives a reply from the storageunit. If the storage unit indicates that the request is rejected, asdetermined in step 506, a revised estimated time is received with themessage in step 508. This revised estimated time may be used to updatethe buffer list in step 510 from which buffers are selected. Processingreturns to step 500 to select another buffer. A buffer for which thesegment is on the same storage unit as the previously selected segmentprobably should not be selected. If the storage unit otherwise acceptsthe request, the data ultimately is received in step 518.

The process from the point of view of the storage unit will now bedescribed in connection with FIG. 21. The storage unit receives arequest from a client in step 520 indicating waiting time E3. If thedata is not yet available in the buffers at that storage unit, asdetermined in step 522, the storage unit rejects the request in step 524and computes a revised estimated time which is sent to the client. Ifthe data is otherwise available and the network connection of thestorage unit is not busy, as determined in step 526, then the clientbecomes an Aactive clients and the communication channel is granted bythe storage unit in step 528, allowing data to be transferred. If thenetwork connection of the storage unit is busy transferring data toanother client, the storage unit maintains a request from a Awaitingclient,≅ to which data is transferred after the data transfer for theAactive clients is completed. In order to determine whether the currentclient should be the Awaiting client,≅ the storage unit estimates a timeby which the transfer could occur, in step 530, based on the number ofrequests with earlier deadlines in the network queue multiplied by thenetwork transmission time for each request. If the computed estimatedtime of availability is greater than the waiting time E3, indicating theclient is not willing to wait that long, as determined in step 532, therequest is rejected in step 524. Also, if the specified priority of thisrequest is lower than the priority for any current waiting client, asdetermined in step 534, the request is rejected in step 524. Otherwise,the request from any current waiting client is rejected in step 536 andthis new client is designated as the current waiting client. When atransfer to the active client is completed, the waiting client becomesthe active client and the data is transferred.

In order to transfer data from a client to a storage unit, a similarprocess may be used for scheduling the network transfer and fortransferring the data from a buffer in the storage unit to nonvolatilestorage. From the point of view of the client, this process will now bedescribed in connection with FIG. 22. This process may be used toimplement step 124 and 126 in FIG. 3.

Unlike the process of reading in which the client may place data into anarbitrary point within its set of buffers, the data to be transferred toa storage unit typically comes from a read pointer from a set of buffersused by the capture system. The capture system typically produces one ormore streams of video information as well as one or more streams ofaudio information. Accordingly, the capture system may select one of thedata streams according to the amount of free buffer space in the streamto receive captured data. This buffer at the current read pointer of theselected stream is selected in step 600. A write request is then sent tothe storage unit in step 602. The request includes an identifier for thesegment, a due time or other priority value, and a threshold E4indicating an amount of time the client is willing to wait. The due timeis used by the storage unit to prioritize network transfer requests. Thethreshold E4 is used by the client, similar to threshold E3 discussedabove, to permit the client to efficiently schedule its own operations.The client, after sending the request to the storage unit, eventuallyreceives a reply in step 604. If the reply indicates that the writerequest was rejected, as determined in step 606, the reply will includean estimated time by which the storage unit will be available to receivethe data. This estimated time, as determined in step 607, may be used bythe client to schedule other operations. If the storage unit accepts therequest to write the data, the client then sends, in step 608, a portionof the segment of the data to the storage unit. A reply may be receivedin step 610 indicating whether or not the write request was successful,as analyzed in step 612. A failure may involve recovery processes instep 614. Otherwise the process is complete as indicated in step 616.

From the point of view of the storage unit, the storage unit receivesthe write request from the client in step 620. The request will indicatea due time or other priority stamp which is used to place the requestwithin the network queue. The storage unit then determines in step 622if a buffer is available for receiving the data. The storage unit maymake such a buffer available. In the unlikely event that no buffers areavailable, the request may be rejected in step 624. Otherwise, a requestis put in the network queue in step 626 indicating the buffer allocatedto receive the data, its priority stamp, and other information about thetransfer. Next, the storage unit determines if the network connection isbusy in step 628. If the network connection is not busy, the storageunit accepts the request in step 630 and sends a message to this effectto the client. The client then transfers the data which is received bythe storage unit in step 632 and placed in the designated buffer. If thedesignated buffer is now full, as determined in step 634, the buffer isplaced in the disk queue with an appropriate priority stamp in step 636.The storage unit=s processing of its disk queue will eventually causethe data to be transferred from the buffer to permanent storage.Otherwise, the storage unit waits until the client sends enough data tofill the buffer as indicated in step 638.

If the network connection of the storage unit is busy, as determined instep 628, the storage unit computes, in step 640, an estimated time bywhich the network connection of the storage unit should be available. Ifthis computed time is greater than the indicated waiting time E4, asdetermined in step 642, the request is rejected in step 624 with anestimate of the time of availability of the storage unit. If the storageunit expects to be able to transfer the data within the waiting time E4indicated by a client, the storage unit compares the priority of therequest with the priority of a request for any currently waiting client,in step 644. If this request is of a lower priority than the request ofthe currently waiting client, the request is rejected. Otherwise, therequest from the currently waiting client is rejected, and this newrequest is made the next request to be processed in step 646.

By scheduling data transfers over the network and by distributing theload on the storage units with selected access to randomly distributedcopies of segments of data, this system is capable of efficientlytransferring multiple streams of data in both directions betweenmultiple applications and multiple storage units in a highly scalableand reliable manner, which is particularly beneficial for distributedmultimedia production.

One application that may be implemented using such a computer network isthe capability to send and return multiple streams to other externaldigital effects systems that are commonly used in live production. Thesesystems may be complex and costly. Most disk-based nonlinear videoediting systems have disk subsystems and bus architectures which cannotsustain multiple playback streams while simultaneously recording aneffects return stream, which limits their abilities to be used in anonline environment. Using this system, several streams may be sent to aneffects system, which outputs an effects data stream to be stored on themultiple storage units. The several streams could be multiple camerasources or layers for dual digital video effects.

It is also possible to have multiple storage units providing data to oneclient to satisfy a client=s need for a high bandwidth stream of datathat has a higher bandwidth than any one storage unit. For example, ifeach of twenty storage units had a 10 MB/s link to a switch and a clienthad a 200 MB/s link to the switch, the client could read 200 MB/s fromtwenty storage units simultaneously, permitting transfer of a datastream for high definition television (HDTV), for example.

Using the procedures outlined above, storage units and clients operateusing local information and without central configuration management orcontrol. A storage unit may be added to the system during operationwithout requiring the system to be shut down. The storage unit simplystarts operation, informs clients of its availability, and thenestablishes processes to respond to access requests. This expandabilitycomplements the capability and reliability of the system.

Having now described a few embodiments of the invention, it should beapparent to those skilled in the art that the foregoing is merelyillustrative and not limiting, having been presented by way of exampleonly. Numerous modifications and other embodiments are within the scopeof one of ordinary skill in the art and are contemplated as fallingwithin the scope of the invention as defined by the appended claims andequivalents thereto.

What is claimed is:
 1. A data storage system, comprising: a plurality ofclient systems, each client system having a file system through whichapplications executed on the client system access data; a plurality ofstorage servers coupled to the plurality of client systems via acomputer network, each storage server storing data accessible throughthe file system; wherein the stored data is divided into segments thatare stored across the plurality of storage servers, with two or morecopies of each segment being distributed among the plurality of storageservers, such that each segment is stored on at least two of the storageservers and wherein the segments of data are distributed nonsequentiallyamong the plurality of storage servers; and storage configured to storeinformation indicating the storage servers on which the segments of thestored data are stored; at least one of the client systems beingconfigured to: access, before reading data, the stored informationindicating the storage servers on which the segments of the data arestored, and communicate directly with the storage servers to request thesegments of the data using the accessed information; and each storageserver being configured to: maintain data defining information thatassociates, for each segment stored on the storage server, theidentifier of the segment with the location of the segment in storage;in response to a request from one of the client systems for a segment,wherein the received request includes the identifier of the requestedsegment, determine the location of the segment in the storage using theinformation that associates the identifier of the requested segment withthe location of the segment in the storage; retrieve the requestedsegment from the storage; and send the retrieved segment over thecomputer network to the client system that requested the segment.
 2. Thedata storage system of claim 1, wherein the plurality of storage serversare configured to respond to requests received from client systems byaccessing the requested segments of the data and sending the requestedsegments of the data directly to the requesting client system.
 3. Thedata storage system of claim 1, wherein the stored information includesa central database accessible by each of the plurality of clientsystems.
 4. A data storage system, comprising: a client system having afile system through which applications executed on the client systemaccess data; a plurality of storage servers coupled to the client systemvia a computer network, each storage server storing data accessiblethrough the file system, wherein the stored data is divided intosegments that are stored across the plurality of storage servers, withtwo or more copies of each segment being distributed among the pluralityof storage servers, such that each segment is stored on at least two ofthe storage servers and wherein the segments of data are distributednonsequentially among the plurality of storage servers; and storageconfigured to maintain information indicating the storage servers onwhich the segments of the stored data are stored; the client systembeing configured to: access, before reading data, the storage to obtainthe information indicating the storage servers on which the segments ofthe data are stored, and communicate directly with the storage serversto request the segments of the data using the accessed information; andeach storage server being configured to: maintain data defininginformation that associates, for each segment stored on the storageserver, the identifier of the segment with the location of the segmentin storage; in response to a request over the computer network from theclient system for a segment, wherein the received request includes theidentifier of the requested segment, determine the location of thesegment in the storage using the information that associates theidentifier of the requested segment with the location of the segment inthe storage; retrieve the requested segment from the storage; and sendthe retrieved segment over the computer network to the client system. 5.The data storage system of claim 4, wherein the catalog manager includesa central database accessible by the client system.
 6. The data storagesystem of claim 4, wherein the plurality of storage servers areconfigured to respond to requests received from client systems byaccessing the requested segments of the data and sending the requestedsegments of the data directly to the requesting client system.
 7. Thedata storage system of claim 4, wherein the client system is furtherconfigured to serialize the segments of data read from the selectedstorage units.
 8. The data storage system of claim 7, wherein the clientsystem includes an application, wherein the application is configured toaccess the stored data on the storage servers through the file system.9. The data storage system of claim 8, wherein the client system isfurther configured to provide the serialized data from the file systemto the application.
 10. The data storage system of claim 4, wherein theclient system comprises a buffer including a plurality of independentlyaccessible portions, and wherein the file system is configured to insertsegments of data received from the storage servers into theindependently accessible portions of the buffer, and wherein anapplication is configured to read data from the buffer sequentially. 11.A data storage system, comprising: a client system having a file systemthrough which applications executed on the client system access data; aplurality of storage servers coupled to the client system via a computernetwork, each storage server storing data accessible through the filesystem, wherein the stored data is divided into segments that are storedacross the plurality of storage servers, with two or more copies of eachsegment being distributed among the plurality of storage servers, suchthat each segment is stored on at least two of the storage servers andwherein the segments of data are distributed nonsequentially among theplurality of storage servers; and storage configured to maintaininformation indicating the storage servers on which the segments of thestored data are stored; the client system being configured to: access,before reading data, the storage to obtain the information indicatingthe storage servers on which the segments of the data are stored, andcommunicate directly with the storage servers to request the segments ofthe data using the accessed information; each storage server beingconfigured to: maintain data defining information that associates, foreach segment stored on the storage server, the identifier of the segmentwith the location of the segment in storage, in response to a requestover the computer network from the client system for a segment, whereinthe received request includes the identifier of the requested segment,retrieve the requested segment from the storage using the informationthat associates the identifier of the requested segment with thelocation of the segment in the storage, and send the retrieved segmentover the computer network to the client system; and the client systembeing further configured to serialize the segments of data read from theselected storage units.
 12. The data storage system of claim 11, whereinthe client system includes an application, wherein the application isconfigured to access the stored data on the storage servers through thefile system.
 13. The data storage system of claim 12, wherein the clientsystem is further configured to provide the serialized data from thefile system to the application.
 14. A data storage system, comprising: aclient system having a file system through which applications executedon the client system access data; a plurality of storage servers coupledto the client system via a computer network, each storage server storingdata accessible through the file system, wherein the stored data isdivided into segments that are stored across the plurality of storageservers, with two or more copies of each segment being distributed amongthe plurality of storage servers, such that each segment is stored on atleast two of the storage servers and wherein the segments of data aredistributed nonsequentially among the plurality of storage servers; andstorage configured to maintain information indicating the storageservers on which the segments of the stored data are stored; the clientsystem being configured to: access, before reading data, the storage toobtain the information indicating the storage servers on which thesegments of the data are stored, and communicate directly with thestorage servers to request the segments of the data using the accessedinformation; each storage server being configured to: maintain datadefining information that associates, for each segment stored on thestorage server, the identifier of the segment with the location of thesegment in storage, in response to a request over the computer networkfrom the client system for a segment, wherein the received requestincludes the identifier of the requested segment, retrieve the requestedsegment from the storage using the information that associates theidentifier of the requested segment with the location of the segment inthe storage, and send the retrieved segment over the computer network tothe client system; and the client system further comprising a bufferincluding a plurality of independently accessible portions, and whereinthe file system is configured to insert segments of data received fromthe storage servers into the independently accessible portions of thebuffer, and wherein an application is configured to read data from thebuffer sequentially.